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Dr. Bruno Manno 

Acting Assistant Secretary for 

Educational Research and Improvement 
U.S. Department of Education 
555 New Jersey Avenue, N.W., Suite 600 
Washington, D.C. 20208-5500 

Dear Bruno: 

On behalf of the Laboratory Review Panel, I am pleased to submit 
to you our report on the pending rfecompetition of the regional 
laboratories . 

Over the course of the past two years, the Panel has reviewed 
many aspects of the laboratory program. In this document we have 
attempted to bring together our reactions, observations and 
recommendations regarding the forthcoming recompetition. 

On behalf of the Panel, I would like to express thanks to the 
OERI staff, who assisted us so ably in our efforts. Although our 
views were represented in the final report, we were provided with 
every level of support and inspiration by a most dedicated staff, 
including David Mack, Barbara Lieb-Brilhart, Joyce Stern and the 
institutional monitors. 

We express special thanks to Milt Goldberg for his support in 
creating^the panel and his continuing interest in our work. We 
especially thank Charles Stalford for his dedication and energy in 
seeing that we kept on track, got it right and did what we were tasked 
to do. Without question, Charles was indispensable. 

We hope that this report will assist you, Nelson Smith, and the 
OERI staff on your plan for the 1990 recompetition . We would be 
pleased to meet with you, to discuss our report, at your convenience. 



Christopher T. Cross 
Vice Chairman 




cc: 



Nelson Smith 
David Mack 
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INFORMATION ABOUT THE PANEL 



The Laboratory Review Panel (LRP) is an external advisory croup established in 1987 by 
the Assistant Secretary for the Office of Educational Research and Improvement (OERI) of 
the U.S. Department of Education. 

The panel's fiut task was to comment on an external review of individual laboratories, 
sponsored by OERI, in the summer of 1987. The panel submitted a report to the Assistant 
Secretary about the review on October 6, 1987 ("Report of the Laboratory Review Panel 
on the 1987 Review of Laboratories"). This report discussed several program- wide issues 
raised by the individual laboratones' external reviews. 

The panel has continued to meet since then, and has been asked to provide advice to OERI 
regarding policies that should be incorporated in the laboratory program through the 
recompetition of existing contracts, which expire in November 1990. Policies incorporated 
in and through the recompetition will guide the program through the period 1990-95. 

The members of the panel are as follows: 

Mr. Christopher Cross (Chair), Vice Chairman and Director, Macro Systems, Inc., Silver 
Spring, Maryland 

Dr. Joy Frechtling, Acting Director, Department of Educational Accountability, Montgomery 
County Public Schools, Rockville, Maryland 

Dr. Ernest House, Professor, Laboratory for Policy Studies, University of Colorado, Boulder 

Dr. Alexander Law, Elk Grove, California (retired from position as Director, Program 
Evaluation and Research Division, California State Department of Education) 

Dr. GarTy McDaniels, President, Softwriters Development Corporation, Linthicum, Maryland 

Dr. Carl Sewell, President, Educative Systems Development Corporation, Plair.field, New 
Jersey 

The panel is solely responsible for the content of this report. No official endorsement by 
OERI or the Department of Education should be inferred. Also, views expressed herein 
are those of the individual panel members and do not necessarily reflect those of the 
organizations with which they are affiliated. 
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1. INTRODUCTION 



The panel believes that periodic recompetition in the lab program is valuable. In 
accordance with "free market" philosophy, the panel is convinced that competition provides 
an opportunity for others to propose new and better ways to operate laboratories. The 
prospect of recompetition also serves to stimulate delivery of higher quality services and 
products by existing holders of lab awards. 

The panel has been asked to make recommendations to the Assistant Secretary of OERI 
regarding policy for the pending laboratory recompetition. With this end in mind, this 
report addresses what the panel believes to be the most critical issues OERI must address 
in the recompetition. For each set of issues, it offers recommendations regarding desirable 
OERI policy. The panel's recommendations are stated at the beginning of each section of 
the report and for the reader's convenience, are also restated separately in Appendix A. 

The panel met on February 23-24, 1989 to formulate its recommendations. In formulating 
these recommendations, the panel has drawn upon a wide variety of inputs. These were 
policy papers commissioned by OFRI to help guide recompetition planning (listed in 
Appendix B), staff papers, the panel's prior experience with the program, and a session 
with laboratory representatives on February 24. 

In its February 24 session with representatives of lab governing boards and executive 
directors, which was open to the public, the panel engaged in a productive discussion about 
the current status and operations of the program. Because policy in a pending OERI 
procurement was involved, Department of Education officials had ruled that the panel could 
not discuss the recompetition, or the future of the program with the representatives in this 
session. 

In the remainder of its February 23-24 meeting, the panel met with OERI staff and 
formulated its recommendations for this report. These discussions were closed to the 
public. 

2. ISSUES AND PANEL RECOMMENDATIONS 
A. Laboratory Linkages within the R&D System 

The panel recommends that OERI commission a high-level and wide-ra nging review 
of the characteristics and productivity of the present R&D system in education. 

At a more immediate level, the panel recommends that OERI critically reexamine the 
relationships within its own network of programs. The panel specifically r ecommends 
that: 
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(a) OERI immediately reexamine the relationships among curren t research, 
development and dissemination activities of its institutional R&D 
programs (labs, centers. Educational R gsnnrn* Information Centers 
(ERIC) and the National Diffusion Network (NDN) before finalizing 
specifications for the pending recompetitjpn of labs and centers ; 

(b) OERI require labs to have center representatives on their governing 
boards and centers to have lab representatives on their advisor y boards; 

(c) OERI appoint a high level advisory committee of represe ntatives of labs, 
centers. ERIC and the NDN to help it gain better c oordination of and 
more productivity from the various parts of its R&D network; 

(d) Department of Education staff representing non-OERI assistance 
programs be invited to participate in the work of this advisory 
committee; 

(e) OERI conduct an annual forum for representatives of its major 
programs to share information about new developments, successes and 
failures, and any other information that would enhance the productivity 
of its R&D network. 



High Level Review of the Educational R&D System - Most of the panel's 
recommendations deal with the lab program itself. And most of them are framed to 
recognize the reality of the context for the lab recompetition. For example, the panel 
recognizes present budget constraints upon labs and other educational programs. 

However, the panel believes that commissioning a thorough reexamination of the 
educational R&D system itself is the most important thing OERI can do, not only because 
of the competition, but also because of its investment in other institutional R&D programs. 

The present R&D system of labs, centers, and other institutional programs has evolved in 
numerous and not always logically consistent ways since those programs were established 
in 1965-66. At that time, with the passage of the Elementary and Secondary Education 
Act (ESEA), the Federal Government vastly increased its support for education, including 
the creation of the lab and center network. 

Among major changes since 1965-66 that should affect current thinking about the 
educational R&D system are the following: 
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o Federal support for large scale curriculum development (in which some labs 
were significantly involved) has ended. 

o There has been an increase in the number and variety of R&D-based 

assistance organizations funded outside the Department of Education or its 
predecessor (HEW). These include educational service agencies and other 
State programs; private sector firms; and agencies not necessarily perceived 
as "educational, 11 but which nonetheless provide significant educational 
services (such as the National Endowments for the Arts and Humanities). 

o There are now other programs for dissemination and technical assistance 
funded by the Department of Education. These include ERIC and NDN 
funded by OERI and targeted assistance programs, such as Technical 
Assistance Centers for Chapter I programs (compensatory education) and 
Regional Resource Centers for P.L. 94-142 programs (special education). 

o States have recently assumed a much stronger leadership role in educational 
reform. 

o Sweeping demographic changes in the nation's student population are 
affecting tf>e agenda for educational R&D. 



Further, some of the assumptions on which the labs were built have proven to be false. 
For example, one assumption was that labs would function as large national organizations, 
on the model of the Brookhaven Laboratory. But no similar approach to massive funding 
of national labs (or a reasonable counterpart in the context of educational R&D) ever 
materialized in the lab program. And, of course, the educational laboratories evolved into 
regional organizations rather than national ones. 

The panel's own sense that a review is needed was reinforced by Dr. James Guthrie. He 
recommends a high-level review of the R&D system in his policy paper. As Guthrie 
states, lacking such a review, alternative recommendations for lab policy in 1990-95 are 
essentially "interim" ones, even if productive. 

Such a review would look at the number, complexity and interrelationships of the various 
parts of the current system. It would ask whether the parts cculd be made to fit together 
more productively. The panel believes the review might also ask whether there are an 
appropriate number and variety of institutional groups in existence now to conduct, or be 
supported in conducting, educational R&D. 

The National Academy of Sciences - OERI should consider contracting with an 
organization such as the National Research Council of the National Academy of Sciences 
(NAS) for such a review. This organization has usefully conducted similar high-level 
reviews for OERI in the past, most recently one focused cn the OERI's National Center for 
Educational Statistics (NCES). 
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There are several advantages to having an outside group conduct the review. One is that 
the impartiality and credibility of the review would be enhanced. Also, many of the finest 
thinkers about educational R&D in the country would likely be willing to participate. The 
pa**el recommends that participants in the review include representatives of labs, centers 
and the other programs direcdy involved, plus their constituents, as well as researchers, 
policy analysts, and representatives of State and local education interests. 

Such a review would take some time to commission and conduct. It would not, therefore, 
interfere with conduct of the present rccompetition. But it could shape fundamental 
program policies in the years ahead. A valuable by-product of the review would be the 
obtaining of more information from the field about the value of various R&D organizations 
to their constituents than is presently available. 

Labs and Centers: Immediate Considerations - The fundamental assumption that 
appears to underlie OERI's thinking about centers and labs is that the former conduct 
research and the latter disseminate research and provide assistance with its use. This 
notion presupposes a linear R&D process (i.e., that research precedes development and 
dissemination, etc.) that does not necessarily exist in the field. The linear notion also 
presupposes a fairly complete separation of research from development and dissemination 
that does not necessarily exist. Indeed, centers conduct a good deal of dissemination, and 
labs conduct some applied research. Further, some labs have indicated their desire to place 
more emphasis on longer term applied research and development, as they have in the past. 

OERI assumes that labs and centers will collaborate in complementary (i.e.synergistic) 
fashion in the performance of their respective functions. In fact, a good deal of un- 
heralded collaboration is occurring, not only labs and centers, but with ERIC and NDN as 
well. Nonetheless, the panel has sufficient evidence from its various inquiries since 1987 
to believe that more can and should be done to improve the relationships between labs and 
other programs funded by OERI and hence the productivity of OERI's own network in the 
R&D system. (The panel recommended that OERI put more effort into fostering 
collaboration between labs and other departmental programs in its 1987 report.) 

The panel's recommendations in this section are intended to strike a balance between 
actions that can have immediate impact (the review of OERI's own network, etc.) and those 
which will have a longer-term impact (the high-level review). 
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B. Future Laboratory Mission 



The panel recommends that, consistent with legislative directio n, OERI reformulate 
the mission for the lab program to provide greater clarity a nd focus in the coming 
recompetition. 

The present mission of the laboratory program is not sufficiently clear. 

Part of the difficulty is due to lack of clarity about presumed relationships among labs, 
centers and other components of the R&D system, as discussed in the previous section. 

Pan of the problem may be the way in which goals of the program are stated in the 
present OERI guidelines. The 1985 OERI Request for Proposals stated that labs should 
"focus on school and classroom improvement" When interpreted broadly, school and 
classroom improvement can encompass almost all types of educational R&D. Those 
guidelines do not provide adequate operational direction about what labs should and should 
not do. 

Program Delivery Strategy and the Mission - An additional factor affecting the lack of 
clarity of the program's mission is that a substantial portion of the labs' resources are 
directed by OERI guidelines to be delivered via an indirect service strategy. 

A nrominent part of this strategy is that the labs are to work "with and through" existing 
organizations to improve schools and classrooms "through R&D-based strategies. Pursuant 
to the indirect service strategy, labs participate in strengthening others' organizational 
support systems (e.g., through capacity-building or training of trainers) and work with 
various intermediaries in a large number of ways, rather than working directly with schools 
and classrooms. 

On the one hand, the degree of client-responsiveness generaled by labs' cooperative efforts 
under the indirect service strategy is good. On the other hand, this strategy leads to a 
situation where the distinctive mission of the program tends to get blurred. (The strategy 
is discussed in more detail in Section C below.) 

Contextual Factors Affecting the Lab Mission - Any refinement of the labs' mission 
must obviously take into account statutory requirements for the program. The OERI 
authorizing legislation requires that regional agendas set by lab governing boards be 
consistent with OERI's "priority research and development needs." (The first two of these 
are increasing student achievement and providing equal educational opportunity.) 

In addition, the legislation specifies that applicants for labs and centers will conduct a 
specified range of activities, including research and development, dissemination and 
technical assistance and that the activities will be consistent with OERI's research and 
development program However, none of these statutory requirements appears to the panel 
to constrain possible refinement of the lab program mission. 
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One contemporary contextual issue is the extent to which the lab mission should be driven 
by the needs of chief state school officers or other State officials in the current 
environment of State-led educational reform. 

In a policy paper written for OERI, David Hornbeck, the former chief State school officer 
in Maryland, suggests labs should be much more responsive to the needs of the chiefs. 
Conversely, in its discussion of the present status of the program with lab representatives, 
the panel was told that the present emphasis upon the needs of chief state school officers 
was about right and that if anything, labs should be more able to meet to the needs of 
other constituencies in their regions. While the role of the chiefs is a significant issue, the 
panel does not believe a mission statement for the entire program need?; to force an "either- 
or position on that issue to be viable. 

Characteristics of a Better Mission Statement - No one mission statement will resolve 
all ambiguities about what the labs should do, but the panel does believe some refinement 
of the program's mission statement should be undertaken. 

In principle, a mission statement should be broad and focus on outcomes rather than 
processes. In this case, the panel recommends a two-part structure for the lab program 
mission. The first part would be a brief and general statement of what the labs should do 
at the program level. The second part would be the presentation of clear and measurable 
objectives by individual labs for their regions. For example, a lab should be expected to 
state (a) what educational problems it expects to focus on, (b) what bodies of R&D-based 
knowledge it expects to use in its work, and (c) the specific benefits it expects to be 
realized within tne 5 year period of its award. 

For illustrative purposes, the panel offers two alternative mission statements at the first 
(general) level in Appendix C. 

(The panel realizes that some contextual factors will inevitably change during the course of 
a 5 year lab award. However, OERI can require applicants in the recompetition to propose 
objectives that are sufficiently specific to be critically reviewed on a comparative basis, 
while at the same time acknowledging the need for flexibility in planning that may arise 
during the course of the 5 year awards.) 



C. Strategy for Delivery of Services 

The panel recommends that OERI revise its approach to the indire ct service strategy 
in the recompetition. Labs should be allowed more flexibility in their choice of 
strategies to serve their regions. 

In the recompetition. applicants' choice of service delivery strateg ies should be judged 
bv the promise they have for helping to attain stated regional objectives. 
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The present OERI policy for delivery of lab services, particularly the indirect service 
strategy discussed briefly above, is perhaps one of two or three topics that have mosi 
occupied the panel since its creation in 1987. (Another is the lab mission, just discussed ) 

Labs are primarily oriented to dissemination and assistance strategies under present OERI 
guidelines. As indicated in the previous section, labs are to work "with and through" 
existing groups, such as education service agencies and State-level decisionmakers, to 
improve schools and classrooms under these guidelines. 

(Some lab work is not subject to this strategy. In addition, OERI staff believe the labs 
sometimes mis understand the indirect service strategy, particularly the extent to which labs 
may actually work with local school districts and schools. Labs can work directly with 
individual school districts, schools and classrooms under a number of circumstances. These 
include cases where there are few assisting organizations with whom a lab can work and 
where schools and classrooms wish to serve as pilot sites for products or services the labs 
have developed. In general, however, OERI's guidelines for work towards school and 
classroom improvement emphasize working with and through others.) 

OERI's rationale for implementing the indirect service strategy in 1985 was straightforward. 
With some 16,000 school districts in the country, the nine labs lacked the resources to 
work directly with a significant number of them. 

As a related matter, the 1985 OERI guidelines re-oriented the labs' work in producing 
R&D-based products (guidelines for instruction, staff development, summaries of research, 
etc.) After 1985, labs' product development was focused on short-term projects that 
assisted their dissemination and assistance strategies, rather than longer-term and more 
ambitious projects (e.g., development of an entire basic skills program) that some of the 
labs had engaged in before. 

Lab Viewpoints - In the open session at the last panel meeting, lab representatives spoke 
about both advantages and problems of the indirect service strategy. One representative 
said that the strategy did not represent much of a change from what his lab had been 
doing previously. Another's view was tha>, the strategy was helpful in encouraging the labs 
to expand their involvement with other agencies in the region. 

Among the problems cited by lab representatives were that they had to take long periods 
of time to establish working relationships with organizations that did not lead to any 
discernible result, and that much of the work under the strategy either diluted the lab's 
impact or showed unimpressive or fragmented results. 

Reflecting on its accumulated perceptions of this topic, the panel believes OERI has gone 
too frr in its present requirements for the indirect service strategy. Even with exceptions, 
the present guidelines ttnd to place the strategy in a "first among equals" status with other 
possible strategies for service delivery. 

On the other hand, the strategy does have a place in the array of ways in which labs might 
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work with regional constituencies. The panel believes tH issue is one of balance, rather 
than either-or. The thrust of the panel's recommendations in this ar*a is therefore to 
provide the labs with more flexibility. 

D. Program Evaluation 

The panel recommends that the emphasis in lab self assessment in the next award 
period be on evaluation to improve program operations (formative evaluation). Such 
evaluation should address the questions "Was the program implemented as agreed* 
upon?" and "How well was it implemented?" 

If lab self-assessment focuses on evaluation for self-improvement in the future, there 
will be an unacceptable gap in OERTs capacity to develop summative information 
about the impact of the program for purposes of accountability and policymaking. 
The panel therefore recommends that QERI request the Congress to provide sufficient 
funds in the future to allow OERI to conduct the necessary summative evaluation. 

Three basic types of program evaluation questions can be asked: (1) Was the program 
implemented as agreed-upon? (2) How well was the program implemented? (3) What 
happened as a result of the program being implemented? 

Lab self-assessments have generated a good deal of information in response to the first two 
questions, but relatively little in response to the third (impact). One lab director directly 
told the panel that his lab was capable of doing the type of study that would provide 
information about impact but preferred to put the money into regional services. 

The panel' s work har been hampered by OERTs own lack of funds to conuuet impact 
evaluations. OERI has developed a comprehensive plan for evaluation of labs during this 
contract period, but has had available only a fraction of the funds necessary to carry it out. 

Evaluative information which has been made available to the panel, notably through the 
OERTs 1987 external review of individual labs, has focused largely on process (questions 1 
and 2 above). In fact, on that dimension, the labs appear to rate quite highly. But the 
panel has no conclusive Information about the impact, or benefits to the country, from the 
lab program. 

The panel does not believe it is realistic to expect the labs to provide conclusive evidence 
about impact. Their concerns tend towards information to improve operations. Further, 
there is an inherent issue of credibility of findings about impact from a lab (or any other 
organization) derived from its self evaluation. The panel's recommendations therefore 
suggest an appropriate division of labor between the labs and OERI for evaluation. 
Unfortunately, the present budgetary climate is not encouraging insofar as OERTs ability to 
do impact studies in the foreseeable future is concerned. The panel's second 
recommendation is intended to address this situation. 
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E. Equitable Distribution of Funds 

The panel recommends tiiat OERI take feasible steps to reduce the g laring inequities 
in the existine perjjupjj funding capacity of labs. 

o OERI should request the Congress to incorporate p resent vear-to-vear 
appropriations for the lab rural initiative into continuing fun ding of the 
program's institutional base. OERI should then m ake those funds 
available for all program activities and re»allocate them am ong the labs 
in a way to reduce the regional funding inequities. 

o OERI should consider the possibility that any re-all ocation of funds 

brought about by the competition for a lab in the Pacific B asin Region 
in 1990 mav provide a further, although smaller, way to reduce region al 
funding inequities. 



The panel has reviewed alternative OERI approaches to the allocation of funds to lab 
regions in the 1985 recompetition. (OERI examined this matter systematically in the 1985 
recompetition. For the first time, non-overlapping boundaries were established for lab 
regions in which all areas of the counjy, including Hawaii and the Pacific territories, were 
served.) 

Establishing an equitable basis for regional distribution of lab funds is a complex process. 
Among the many factors to consider are the number of States, the size of the elementary 
and secondary student population, and the geography and population density of the region. 
In addition, six of the present nine labs have been in existence for more than two decades. 
As a result, there are political factors present, based on historical precedents and 
expectations regarding the level of funding for these labs. These expectations are held by 
legislators and constituents, as well as the labs themselves. 

OERI's approach to equitable funding in 1985 involved three specific factors: (a) a more 
less fixed amount OERI judged necessary for any lab to support its institutional functions 
of organization, management and governance; (b) the number of States in the region; and 
(c) the population of elementary and secondary school students in public and private 
schools in the region. 

Per-Student Spending Capacity - OERI sought a workable accommodation to these 
factors in 1985. The panel does not criticize the 1985 funding allocations. But one aspect 
of the resulting allocations needs attention in the pending recompetition. 

The 1985 funding allocations deri\ed from L.i three factors above leave considerable 
variance in the capacity for per-student expenditures that the labs are capable of making 
within their regions. In fact, the ratio of the highest per-student expenditure that a lab can 
make to the lowest is roughly 10:1. The panel recognizes that the factors affecting costs 
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arc so complex that full equity is probably not possible. However, there is no justification 
for continuing a disparity of the present magnitude. 

The per-student expenditure calculation is an analytical tool. It does not mean that each 
lab actually spends that amount per-student. Nor is it the only way to view lab 
allocations. However, the per-student figure is a good indicator of the vastly different 
challenges that individual labs face when serving their regions. A lab with a high per- 
student allocation can plan and deliver a more aggressive strategy for peneaxHng its region 
with services and products than one with a low per-student allocation. The latter type lab 
will, taking into account the relatively small percent its funds represent of toM regional 
expenditures for education, be forced into a less central role in its rr.gion. 

"Less central" does not necessarily mean a lab will be inconsequential. (The median 
funding per lab at present, including rural programs, is approximately $2 million). Nor 
does it mean that the lab cannot be useful. It does mean that in most such cases the lab 
will be a smaller fish in a bigger pond. It will likely be obliged to "tag on" on to other 
initiatives, or leverage its services through intermediaries to a greater extent. (This factor 
clearly has implications for how a lab would conceive of its service delivery strategy, as 
discussed earlier ir this report.) 

Possible Steps - "Levelling up" funding of all labs to the per-student level of the "richest" 
one would not be easy to do, and may not be the most equitable formulation. Again, that 
is because student population was only one of three factors in OERI's regional allocations 
in 1985. If weighting of the other two factors (a fixed amount for lab institutional 
functions and the number f States in the region) was held constant, manipulation of 
funding based on the pupil population factor would not produce a direct 1:1 change in the 
overall disparity in per-student capacity that is now observable. 

In any event, "levelling up" funding of labs to equalize their per-student funding capacity 
would be prohibitively expensive. Conversely, "levelling-down" the funding of some labs 
to equalize the per-student allocations would be practically and politically unfeasible. 

The panel does believe that circumstantial factors in the pending recompetition do provide 
a possible way to reduce the levels of inequity in present per-student funding capacity of 
labs. 

The first factor is the current year-to-year appropriations for the lab rural initiative, now in 
its third year. If these funds ($5,138 million in FY 89) could be incorporated into the lab 
program's regular base and made available for all activities, the additional money could be 
re-allocated in a way to reduce the per-student inequities. 

The second factor is the competition which will occur in the Pacific Basin Region for the 
first time in 1990. (This region includes Hawaii and several trust territories.) In 
accordance with a decision made as part of the 1985 recompetition, the Northwest Lab has 
been serving this region while preparations were being made for a separate lab to ■serve the 
region in 1990 and thereafter. 
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It may be necessary to reallocate funds among the '-abs to provide funds for the new 
Pacific Basin Region. If so, an opportunity may exist on a smaller scale to further r 
existing inequities in funding among the regions. 



F. Outside Funding of Laboratories 

The panel recognizes that labs have a legal right to see k outside funds: nevertheless, 
it recommends that OERI review such activities as necessary to e nsure that the labs 
ability to conduct work pursuant to their core regional missio n is not jeopardized. 

The panel further recommends that OERI ensure that its funds a re not used by labs 
to subsidize costs of writing proposals for outside funding or otherwise positioning 
themselves to receive outside funding. 



Tn its 1987 report to ihe Assistant Secretary, the panel recommended that the 
"entrepreneurial" behavior of labs in seeking funds outside those from OERI be studied 
further prior to the recompetition. (At present, OERI believes that labs' outside funding as 
a percent of their total funding on an individual basis ranges from less than 5 percent to 
approximately 50 percent The median probably approximates 15 percent) 

The panel is concerned about acquisition of outside funds for a number of reasons. First, 
these entrepreneurial efforts may divert the labs from their OERI-funded mission. Second, 
the commitment of key personnel to the OERI contract may be lost or reduced through 
diversion to other projects and priorities. Third, as a result of OERI's funding for their 
institutional functions, OERI may unintentionally be subsidizing labs to compete with others 
for outside work in the field. 

Of special concern is assuring that activities in support of preparing proposals for outside 
funding are not being included in the pool of costs used to establish the labs' indirect cost 
rate. (The evidence on this general set of concerns is inconclusive, but numerous outsiders 
have raised the question of whether labs are enabled to compete unfairly with others 
because of their institutional support from OERI.) 

In their meeting with the panel, lab representatives generally defended their work outside 
the OERI lab contracts. Among the arguments they cited were that the outside work 
contributed to the labs' overall capabilities, served to strengthen the R&D "infra-structure," 
and did not pose a threat to OERI-funded work. 

The lab representatives acknowledged some differences among themselves over the issue of 
outside funding in the past. They told the panel, however, that uncertainty of funding in 
recent years has tended to bring them together on the subject. 
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Labs' acquisition of outside funds does has advantages. One is that it makes labs less 
dependent upon OERI funding for survival. Another is that outside funding potentially 
makes labs more sensitive to market forces and needs in their regions. 

The panel understands that OERI's monitoring practices in this area during the current 
contract period have led to three rules of thumb when weighing the desirability of a lab's 
seeking outside funds in specific instances. These rules are that (a) the activities to be 
funded from outside would not be inconsistent with the lab's regional mission; (b) the 
outside funding would not dilute performance on OERI-funded work; and (c) the outside 
funding would not take the lab's resources outside its region. The panel believes that 
OERI should continue to follow this approach in monitoring labs' outside activities in the 
future. 



G, Procurement Mechanism for New Awards 

The panel recommends that the Department of Education ask the Congress to 
consider establishing a new form of procurement instrument that would be better 
suited to the lab program (and possibly other programs as well) than the existing 
ones. Such an instrument should both protect the legal rights of the labs and the 
Government and still allow the labs a reasonable degree of flexibility in planning and 
implementing their programs over the 5 year period of 'he new awards. 

Should such a new instrument not be available, the panel would recommend the use 
of contracts, among the existing alternatives. 

The panel also considered a mix N grants and contracts in the recompetition^ Tt>r 
example, contracts could be used to procure labs' institutional functions and grants 
could be used to assist their regional services. However, the panel recognizes that the 
logistical problems that would be presented by running concurrent competitions may 
be insurmountable. 

In any event, the panel is concerned that the Department of Education is not taking 
maximum advantage of the existing discretionary powers to facilitate contract 
administration that are vested in its lab staff as Contracting Officers' Technical 
Representatives (COTR's), The panel therefore recommends that OERI explore with 
the Department's contracts office ways in which administration of lab contracts could 
be made more efficient and lass burdensome through greater exercise of the COTR's 
powers. 

Contracts as presently used are not well-suited to administration of the lab program. 
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In its 1987 report, the panel expressed its interest in the efficiency and productivity of the 
organizational and contractual relationships between the labs and OERI. It also expressed 
concern about the specificity of contracts: 

The panel believes that tne specificity of contracts emphasizes process at the 
expense of (1) a concept of overall lab role (2) substantive emphasis (3) vision of 
successful service and (4) indicators of success. 

In 1987, the panel further stated a concern that contracts appear to impose an unrealistic 
degree of precision upon labs to state in advance what they plan to do in future years, at 
least beyond the current and succeeding year. 

Contracts are ordinarily used when the Government wishes to purchase something, for its 
own or others* use. Grants, by comparison, are used when the Government wishes to lend 
assistance to others. In the abstract, grants would thus appear well suited to the lab 
program. 

Nonetheless, from the lab viewpoint, there are several significant disadvantages to grants. 
They are as follows: 

o the labs believe the specificity and legally-binding nature of a contract (it is 
signed by both parties, while a grant is only signed by the issuer) help 
prevent arbitrary actions by the funding agency 

o a contractor has the right to appeal the terms of an involuntary termination 
to a contract appeals board 

o under a contract, labs can receive a management fee. 

Ths fee issue has been a troublesome one since 1985. The Federal Acquisition Regulations 
(FAR) allow the use of no-fee contracts for R&D. OERI issued cost-reimbursable, no-fee 
contracts as a result of the 1985 recompetition. Labs would like to have fees. Such fees 
would allow them to finance receivables pending reimbursement of expenses from the 
Government; build up reserves for adverse contingencies, such as disallowance of costs 
through later audits; and be used for institution-building purposes. As indicated in its 1987 
report, the panel is sympathetic to the labs* position regarding the fee issue. 

Aside from the labs* preference for contracts, OERI staff told the panel they have other 
concerns about the use of grants. One is that OERI would have quite limited powers to 
specify the work to be done and the "deliverables" (products) resiiting from it. OERI's 
ability to determine such things would be largely limited to rather general specifications in 
the Department of Education G eneral Administrative Regulations (EDGAR), and OERI 
program regulations, which cover centers as well as labs. 
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The panel briefly explored the possibility of using cooperative agreements in the program. 
These agreements art a special form of a grant. Under them, however, the Government 
participates more closely in the work with the "grantee" and correspondingly assumes a 
greater degree of responsibility for its success or failure. 

Three difficulties with cooperative agreements in the lab program are that (1) assumption 
by OERI of greater responsibility for the programs under the agreements is not consonant 
with the statutory responsibility of regional boards for governance of the labs; (2) it is not 
clear that OERI could bear the fiscal and staffing burden (e.g., of travel) to work closely 
with the labs in the mar ler envisioned by a cooperative agreement; and (3) as a form of 
grant, cooperative agreements would bear no fee. 

In general, the panel is not satisfied with any of the existing procurement options for the 
program in the 1990-95 period; hence, its recommendation that OERI seek a new 
procurement instrument from the Congress. 

H. Means to Facilitate Competition 

The panel recommends that OERI take every feasible step to stimulate actual 
competition in the forthcoming recompetition. 

o O ERI should make publicly available the various papers and studies it 
has commissioned to help its recompetition planning. These include 
policy papers, findings from contracted studies, records of advice from 
any other sources it has solicited, minutes of the panel's meeting with 
lab representatives in February 1989, the panel's 1987 report on the 
external review of individual labs and this report . 

o Notice of the availability of these materials should be made as promptly 
as possible. 

o OERI should allow at least 3 months for response to the solicitation in 
order to facilitate preparation of high quality proposals from many 
sources. 

o OERI should structure the solicitation, particularly the proposal 
evaluation criteria, to accommodate the legitimate needs of 
nonincumbents: for example, by giving due credit for excellence of plans 
and staffing and not over-weighting existing organizational capacity. 

o OERI should not require non-incumbent applicants to gain commitments 
to serve on lab governing boards from chief State school officers and 
similar regional officials kg a condition of their applications. Reasonable 
l ans for obtaining such commitments after the award if a 
nonlncumbent is funded should be acceptable. 

14 



ERJC 



18 



As stated at the beginning, the panel believes competition is valuable for the lab program. 



Conducting a competitive procurement, however, does not guarantee the presence of 
competitors, i.e., more than one applicant for each award per lab region. The panel 
recognizes that formulating a lab proposal is a formidable task. Under present guidelines, 
nonincumbent applicants will have to form a brand new organization, gain numerous 
commitments to participate in a regional governing board, and plan a complex program of 
work. 

A particularly daunting task for nonincumbents applicants is to gain commitments from 
chief state school officers and other significant officials to serve on their governing boards. 
Many if not most of these officials, particularly the chiefs, probably serve on the existing 
labs* governing boards. Nonincumbents are therefore not in a good position to request 
their commitment to another lab, if funded. 

But the 1985 lab recompetition demonstrated that numerous people and organizations not 
currently holding lab contracts could be motivated to compete. The panel believes that 
stimulation of competition by nonincumbents is good for the program as a whole, existing 
labs included, just as vigorous competition in elections is good for the American 
democratic system of government. 

The absence of funds for planning grants, as were used in 1985, will make the task for 
nonincumbents harder. Still, the panel believes that OERI can take vital steps to facilitate 
the preparation of high quality proposals by all applicants and to enable nonincumbents to 
prepare actually competitive proposals. These steps, as described in the panel's 
recommendations include (1) provision of full information to all prospective competitors, 
(2) providing ample time for preparation of proposals and (3) structuring the solicitation to 
accommodate the legitimate needs of non-incumbents. 
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APPENDIX A 
RESTATEMENT OF PANEL RECOMMENDATIONS 

The panel's recommendations are restated below in their entirety for the reader's 
convenience. 

A. Laboratory Linkages within the R&D System 

The panel recommends that OERI commission a high-level and wide-ranging review of the 
characteristics and productivity of the present R&D system in education. 

At a more immediate level, the panel recommends that OERI critically reexamine the 
relationships within its own network of programs. The panel specifically recommends that: 

(a) OERI immediately reexamine the relationships among current research, 
development, and dissemination activities of its institutional R&D programs 
(labs, centers, Educational Resources Information Centers (ERIC) and the 
National Diffusion Network (NDN) before finalizing specifications for the 
pending recompetition of labs and centers; 

(b) OERI require labs to have center representatives on their governing boards 
and centers to have lab representatives on their advisory boards; 

(c) OERI appoint a high level advisory committee of representatives of labs, 
centers, ERIC and the NDN to help it gain better coordination of and more 
productivity from the various parts of its R&D network; 

(d) Department of Education staff representing non-OERI assistance programs be 
invited to participate in the work of this advisory committee; 

(e) OERI conduct an annual forum for representatives of its major programs to 
share information about new developments, successes and failures, and any 
other information that would enhance the productivity of its R&D network. 



B. Future Laboratory Mission 

The panel recommends that, consistent with legislative direction, OERI reformulate the 
mission for the lab program to provide greater clarity and focus in the coming 
recompetition. 
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C. Strategy for Delivery of Services 



The panel recommends that OERI revise its approach to the indirect service strategy in the 
recompetition. Labs should be allowed more flexibility in their choice of strategies to 
serve their regions. 

In the reccaipetition, applicants* choice of service delivery strategies should be judged by 
the promise they have for helping to attain stated regional objectives. 



D. Program Evaluation 

The panel recommends that the emphasis in lab self assessment in the next award period 
be on evaluation to improve program operations (formative evaluation). Such evaluation 
should address the questions "Was the program implemented as agreed-upon?" and "How 
well was it implemented?" 

If lab self-assessment focuses on evaluation for self-improvement in the future, there will 
be an unacceptable gap in OERTs capacity to develop summative information about the 
impact of the program for purposes of accountability and policymaking. The panel 
therefore recommends that OERI request the Congress to provide sufficient funds in the 
future to allow OERI to conduct the necessary summative evaluation. 



E. Equitable Distribution of Funds 

The panel recommends that OERI take feasible steps to reduce the glaring inequities in the 
existing per-pupil funding capacity of labs. 

o OERI should request the Congress 10 incorporate present year-to-year 
appropriations for the lab rural initiative into continuing funding of the 
program's institutional base. OERI should then make those funds available 
for all program activities and re-allocate them among the labs in a way to 
reduce the regional funding inequities. 

o OERI should consider the possibility that any re-allocation of funds brought 
about by the competition for a lab in the Pacific Basin Region in 1990 may 
provide a further, although smaller, way to reduce regional funding 
inequities. 



F. Outside Funding of Laboratories 

The panel recognizes that labs have a legal right to seek outside funds; nevertheless, it 
recommends that OERI review such activities as necessary to ensure that the labs* ability to 
conduct work pursuant to their core regional mission is not jeopardized. 
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The panel further recommends that OERI ensure that its funds are not used by labs to 
subsidize costs of writing proposals for outside funding or otherwise positioning themselves 
to receive outside funding. 

G. Procurement Mechanism for New Awards 

The panel recommends that the Department of Education ask the Congress to consider 
establishing a new form of procurement instrument that would be better suited to the lab 
program (and possibly other programs as well) than the existing ones. Such an instrument 
should both protect the legal rights of the labs and the Government and still allow the labs 
a reasonable degree of flexibility in planning and implementing their programs over the 5 
year period of the new awards. 

Should such a new instrument not be available, the panel would recommend the use of 
contracts, among the existing alternatives. 

The panel also considered a mix of grants and contracts in the recompetition. For 
example, contracts could be used to procure labs 9 institutional functions and grants could be 
used to assist their regional services. However, the panel recognizes that the logistical 
problems that would be presented by running concurrent competitions may be 
insurmountable. 

In any event, the panel is concerned that the Department of Education is not taking 
maximum advantage of the existing discretionary powers to facilitate contract administration 
that are vested in its lab staff as Contracting Officers* Technical Representatives (COTR's). 
The panel therefore recommends that OERI explore v/ith the Department's contracts office 
ways in which administration of lab contracts could be made more efficient and less 
burdensome through greater exercise of the COTR's powers. 



H. Means to Facilitate Competition 

The panel recommends that OERI take every feasible step to stiniuiate actual competition 
in the forthcoming recompetition. 

o OERI should make publicly available the various papers and studies it has 
commissioned to help its recompetition planning. These include policy 
papers, findings from contracted studies, records of advice from any other 
sources it has solicited, minutes of the panel's meeting with lab 
representatives in February 1989, the panel's 1987 report on the external 
review of individual labs and this report. 

o Notice or the availability of these materials should be made as promptly as 
possible. 

o OERI should allow at least 3 months for response to the solicitation in order 
to facilitate preparation of high quality proposals from many sources. 
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OERI should structure the solicitation, particularly the proposal evaluation 
criteria, to accommodate the legitimate needs of nonincumbents; for example, 
by giving due credit for excellence of plans and staffing and not over- 
weighting existing organizational capacity. 

OERI should not require non-incumbent applicants to gain commitments to 
serve on lab governing boards from chief State school officers and similar 
regional officials as a condition of their applications. Reasonable plans for 
obtaining such commitments after the award if a nonincumbent is funded 
should be acceptable. 
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APPENDIX B 

LIST OF POLICY PAPERS COMMISSIONED BY OERI 
FOR RECOMPETITION PLANNING 



OERI has commissioned several policy papers designed to provide different perspectives on 
the lab program and laboratory functions in general to help planning for the recompetition. 

A listing of the authors of the papers and their topics follow. All but the papers by 
Thomas Good and William Dunn were made available to the panel for its February 23-24 
meeting. The other two pppers are being made available to the panel subsequently. 

o Brenda Turnbull, Policy Studies Associates, Inc., "A Ccmparison of Service Modes 
in ED's (the Department of Education's) Technical Assistance Programs 

o David Hornbeck, Johns Hopkins University and formerly chief State school officer 
in Maryland, "The Perspectives of a Former Chief State School Officer" 

o Thomas Good, University or Missouri, Columbia, "Classroom and School Research: 
Investments in Enhancing Schools" 

o Florctta McKenzie, The McKenzie Group, Washington, D.C. and formerly 

Superintendent of Schools in the District of Columbia, "The Future of Regional 
Educational Laboratories in Contributing to Urban School Improvement" 

o James Guthrie, University of California, Berkeley, "Regional Educational 
Laboratories: History and Prospect" 

o Neville Postlethwaite, University of Hamburg, (The Federal Republic of Germany), 
"School and Classroom Improvement in Two European Countries" 

o William Dunn, University of Pittsburgh, "Perspectives on Regional Laboratories from 
Knowledge Transfer in Non-Educational Fields" 
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APPENDIX C 

ILLUSTRATIVE STATEMENTS OF LAB PROGRAM MISSION 



The panel has recommended that the mission statement for labs be structured into two 
parts. The first part would be a general statement of what labs should do at the program 
level. The second part would be of statements of clear and measurable objectives by labs 
for their individual regions. 

Illustrative statements of the first, general mission statement at the program level follow. 

The mission of laboratories is to help education practitioners and policymakers apply 
research-based knowledge for purposes of school improvement. 



-or- 



The mission of educational laboratories is to gain applications of research-based 
knowledge for improvement of educational practice and policy. 
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